2011-07-20 19:51:00 +04:00
/ * bpf_ j i t . S : P a c k e t / h e a d e r a c c e s s h e l p e r f u n c t i o n s
* for P P C 6 4 B P F c o m p i l e r .
*
* Copyright 2 0 1 1 M a t t E v a n s < m a t t @ozlabs.org>, IBM Corporation
*
* This p r o g r a m i s f r e e s o f t w a r e ; you can redistribute it and/or
* modify i t u n d e r t h e t e r m s o f t h e G N U G e n e r a l P u b l i c L i c e n s e
* as p u b l i s h e d b y t h e F r e e S o f t w a r e F o u n d a t i o n ; version 2
* of t h e L i c e n s e .
* /
# include < a s m / p p c _ a s m . h >
# include " b p f _ j i t . h "
/ *
* All o f t h e s e r o u t i n e s a r e c a l l e d d i r e c t l y f r o m g e n e r a t e d c o d e ,
* whose r e g i s t e r u s a g e i s :
*
* r3 s k b
* r4 ,r5 A ,X
* r6 * * * a d d r e s s p a r a m e t e r t o h e l p e r * * *
* r7 - r10 s c r a t c h
* r1 4 s k b - > d a t a
* r1 5 s k b h e a d l e n
* r1 6 - 3 1 M [ ]
* /
/ *
* To c o n s i d e r : T h e s e h e l p e r s a r e s o s m a l l i t c o u l d b e b e t t e r t o j u s t
* generate t h e m i n l i n e . I n l i n e c o d e c a n d o t h e s i m p l e h e a d l e n c h e c k
* then b r a n c h d i r e c t l y t o s l o w _ p a t h _ X X X i f r e q u i r e d . ( I n f a c t , c o u l d
* load a s p a r e G P R w i t h t h e a d d r e s s o f s l o w _ p a t h _ g e n e r i c a n d p a s s s i z e
* as a n a r g u m e n t , m a k i n g t h e c a l l s i t e a m t l r , l i a n d b l l r . )
* /
.globl sk_load_word
sk_load_word :
cmpdi r _ a d d r , 0
2012-04-29 23:02:19 +04:00
blt b p f _ s l o w _ p a t h _ w o r d _ n e g
.globl sk_load_word_positive_offset
sk_load_word_positive_offset :
2011-07-20 19:51:00 +04:00
/* Are we accessing past headlen? */
subi r _ s c r a t c h1 , r _ H L , 4
cmpd r _ s c r a t c h1 , r _ a d d r
blt b p f _ s l o w _ p a t h _ w o r d
/* Nope, just hitting the header. cr0 here is eq or gt! */
2013-09-24 16:13:35 +04:00
# ifdef _ _ L I T T L E _ E N D I A N _ _
lwbrx r _ A , r _ D , r _ a d d r
# else
2011-07-20 19:51:00 +04:00
lwzx r _ A , r _ D , r _ a d d r
2013-09-24 16:13:35 +04:00
# endif
2011-07-20 19:51:00 +04:00
blr / * R e t u r n s u c c e s s , c r0 ! = L T * /
.globl sk_load_half
sk_load_half :
cmpdi r _ a d d r , 0
2012-04-29 23:02:19 +04:00
blt b p f _ s l o w _ p a t h _ h a l f _ n e g
.globl sk_load_half_positive_offset
sk_load_half_positive_offset :
2011-07-20 19:51:00 +04:00
subi r _ s c r a t c h1 , r _ H L , 2
cmpd r _ s c r a t c h1 , r _ a d d r
blt b p f _ s l o w _ p a t h _ h a l f
2013-09-24 16:13:35 +04:00
# ifdef _ _ L I T T L E _ E N D I A N _ _
lhbrx r _ A , r _ D , r _ a d d r
# else
2011-07-20 19:51:00 +04:00
lhzx r _ A , r _ D , r _ a d d r
2013-09-24 16:13:35 +04:00
# endif
2011-07-20 19:51:00 +04:00
blr
.globl sk_load_byte
sk_load_byte :
cmpdi r _ a d d r , 0
2012-04-29 23:02:19 +04:00
blt b p f _ s l o w _ p a t h _ b y t e _ n e g
.globl sk_load_byte_positive_offset
sk_load_byte_positive_offset :
2011-07-20 19:51:00 +04:00
cmpd r _ H L , r _ a d d r
ble b p f _ s l o w _ p a t h _ b y t e
lbzx r _ A , r _ D , r _ a d d r
blr
/ *
net: filter: get rid of BPF_S_* enum
This patch finally allows us to get rid of the BPF_S_* enum.
Currently, the code performs unnecessary encode and decode
workarounds in seccomp and filter migration itself when a filter
is being attached in order to overcome BPF_S_* encoding which
is not used anymore by the new interpreter resp. JIT compilers.
Keeping it around would mean that also in future we would need
to extend and maintain this enum and related encoders/decoders.
We can get rid of all that and save us these operations during
filter attaching. Naturally, also JIT compilers need to be updated
by this.
Before JIT conversion is being done, each compiler checks if A
is being loaded at startup to obtain information if it needs to
emit instructions to clear A first. Since BPF extensions are a
subset of BPF_LD | BPF_{W,H,B} | BPF_ABS variants, case statements
for extensions can be removed at that point. To ease and minimalize
code changes in the classic JITs, we have introduced bpf_anc_helper().
Tested with test_bpf on x86_64 (JIT, int), s390x (JIT, int),
arm (JIT, int), i368 (int), ppc64 (JIT, int); for sparc we
unfortunately didn't have access, but changes are analogous to
the rest.
Joint work with Alexei Starovoitov.
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Mircea Gherzan <mgherzan@gmail.com>
Cc: Kees Cook <keescook@chromium.org>
Acked-by: Chema Gonzalez <chemag@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-05-29 12:22:50 +04:00
* BPF_ L D X | B P F _ B | B P F _ M S H : l d x b 4 * ( [ o f f s e t ] & 0 x f )
2012-04-29 23:02:19 +04:00
* r_ a d d r i s t h e o f f s e t v a l u e
2011-07-20 19:51:00 +04:00
* /
.globl sk_load_byte_msh
sk_load_byte_msh :
2012-04-29 23:02:19 +04:00
cmpdi r _ a d d r , 0
blt b p f _ s l o w _ p a t h _ b y t e _ m s h _ n e g
.globl sk_load_byte_msh_positive_offset
sk_load_byte_msh_positive_offset :
2011-07-20 19:51:00 +04:00
cmpd r _ H L , r _ a d d r
ble b p f _ s l o w _ p a t h _ b y t e _ m s h
lbzx r _ X , r _ D , r _ a d d r
rlwinm r _ X , r _ X , 2 , 3 2 - 4 - 2 , 3 1 - 2
blr
/ * Call o u t t o s k b _ c o p y _ b i t s :
* We' l l n e e d t o b a c k u p o u r v o l a t i l e r e g s f i r s t ; we have
* local v a r i a b l e s p a c e a t r1 + ( B P F _ P P C _ S T A C K _ B A S I C ) .
* Allocate a n e w s t a c k f r a m e h e r e t o r e m a i n A B I - c o m p l i a n t i n
* stashing L R .
* /
# define b p f _ s l o w _ p a t h _ c o m m o n ( S I Z E ) \
mflr r0 ; \
std r0 , 1 6 ( r1 ) ; \
/* R3 goes in parameter space of caller's frame */ \
std r _ s k b , ( B P F _ P P C _ S T A C K F R A M E + 4 8 ) ( r1 ) ; \
std r _ A , ( B P F _ P P C _ S T A C K _ B A S I C + ( 0 * 8 ) ) ( r1 ) ; \
std r _ X , ( B P F _ P P C _ S T A C K _ B A S I C + ( 1 * 8 ) ) ( r1 ) ; \
addi r5 , r1 , B P F _ P P C _ S T A C K _ B A S I C + ( 2 * 8 ) ; \
stdu r1 , - B P F _ P P C _ S L O W P A T H _ F R A M E ( r1 ) ; \
/* R3 = r_skb, as passed */ \
mr r4 , r _ a d d r ; \
li r6 , S I Z E ; \
bl s k b _ c o p y _ b i t s ; \
2012-06-21 21:50:27 +04:00
nop; \
2011-07-20 19:51:00 +04:00
/* R3 = 0 on success */ \
addi r1 , r1 , B P F _ P P C _ S L O W P A T H _ F R A M E ; \
ld r0 , 1 6 ( r1 ) ; \
ld r _ A , ( B P F _ P P C _ S T A C K _ B A S I C + ( 0 * 8 ) ) ( r1 ) ; \
ld r _ X , ( B P F _ P P C _ S T A C K _ B A S I C + ( 1 * 8 ) ) ( r1 ) ; \
mtlr r0 ; \
cmpdi r3 , 0 ; \
blt b p f _ e r r o r ; /* cr0 = LT */ \
ld r _ s k b , ( B P F _ P P C _ S T A C K F R A M E + 4 8 ) ( r1 ) ; \
/* Great success! */
bpf_slow_path_word :
bpf_ s l o w _ p a t h _ c o m m o n ( 4 )
/* Data value is on stack, and cr0 != LT */
lwz r _ A , B P F _ P P C _ S T A C K _ B A S I C + ( 2 * 8 ) ( r1 )
blr
bpf_slow_path_half :
bpf_ s l o w _ p a t h _ c o m m o n ( 2 )
lhz r _ A , B P F _ P P C _ S T A C K _ B A S I C + ( 2 * 8 ) ( r1 )
blr
bpf_slow_path_byte :
bpf_ s l o w _ p a t h _ c o m m o n ( 1 )
lbz r _ A , B P F _ P P C _ S T A C K _ B A S I C + ( 2 * 8 ) ( r1 )
blr
bpf_slow_path_byte_msh :
bpf_ s l o w _ p a t h _ c o m m o n ( 1 )
lbz r _ X , B P F _ P P C _ S T A C K _ B A S I C + ( 2 * 8 ) ( r1 )
rlwinm r _ X , r _ X , 2 , 3 2 - 4 - 2 , 3 1 - 2
blr
2012-04-29 23:02:19 +04:00
/ * Call o u t t o b p f _ i n t e r n a l _ l o a d _ p o i n t e r _ n e g _ h e l p e r :
* We' l l n e e d t o b a c k u p o u r v o l a t i l e r e g s f i r s t ; we have
* local v a r i a b l e s p a c e a t r1 + ( B P F _ P P C _ S T A C K _ B A S I C ) .
* Allocate a n e w s t a c k f r a m e h e r e t o r e m a i n A B I - c o m p l i a n t i n
* stashing L R .
* /
# define s k _ n e g a t i v e _ c o m m o n ( S I Z E ) \
mflr r0 ; \
std r0 , 1 6 ( r1 ) ; \
/* R3 goes in parameter space of caller's frame */ \
std r _ s k b , ( B P F _ P P C _ S T A C K F R A M E + 4 8 ) ( r1 ) ; \
std r _ A , ( B P F _ P P C _ S T A C K _ B A S I C + ( 0 * 8 ) ) ( r1 ) ; \
std r _ X , ( B P F _ P P C _ S T A C K _ B A S I C + ( 1 * 8 ) ) ( r1 ) ; \
stdu r1 , - B P F _ P P C _ S L O W P A T H _ F R A M E ( r1 ) ; \
/* R3 = r_skb, as passed */ \
mr r4 , r _ a d d r ; \
li r5 , S I Z E ; \
bl b p f _ i n t e r n a l _ l o a d _ p o i n t e r _ n e g _ h e l p e r ; \
2012-06-21 21:50:27 +04:00
nop; \
2012-04-29 23:02:19 +04:00
/* R3 != 0 on success */ \
addi r1 , r1 , B P F _ P P C _ S L O W P A T H _ F R A M E ; \
ld r0 , 1 6 ( r1 ) ; \
ld r _ A , ( B P F _ P P C _ S T A C K _ B A S I C + ( 0 * 8 ) ) ( r1 ) ; \
ld r _ X , ( B P F _ P P C _ S T A C K _ B A S I C + ( 1 * 8 ) ) ( r1 ) ; \
mtlr r0 ; \
cmpldi r3 , 0 ; \
beq b p f _ e r r o r _ s l o w ; /* cr0 = EQ */ \
mr r _ a d d r , r3 ; \
ld r _ s k b , ( B P F _ P P C _ S T A C K F R A M E + 4 8 ) ( r1 ) ; \
/* Great success! */
bpf_slow_path_word_neg :
lis r _ s c r a t c h1 ,- 3 2 / * S K F _ L L _ O F F * /
cmpd r _ a d d r , r _ s c r a t c h1 / * a d d r < S K F _ * * /
blt b p f _ e r r o r / * c r0 = L T * /
.globl sk_load_word_negative_offset
sk_load_word_negative_offset :
sk_ n e g a t i v e _ c o m m o n ( 4 )
lwz r _ A , 0 ( r _ a d d r )
blr
bpf_slow_path_half_neg :
lis r _ s c r a t c h1 ,- 3 2 / * S K F _ L L _ O F F * /
cmpd r _ a d d r , r _ s c r a t c h1 / * a d d r < S K F _ * * /
blt b p f _ e r r o r / * c r0 = L T * /
.globl sk_load_half_negative_offset
sk_load_half_negative_offset :
sk_ n e g a t i v e _ c o m m o n ( 2 )
lhz r _ A , 0 ( r _ a d d r )
blr
bpf_slow_path_byte_neg :
lis r _ s c r a t c h1 ,- 3 2 / * S K F _ L L _ O F F * /
cmpd r _ a d d r , r _ s c r a t c h1 / * a d d r < S K F _ * * /
blt b p f _ e r r o r / * c r0 = L T * /
.globl sk_load_byte_negative_offset
sk_load_byte_negative_offset :
sk_ n e g a t i v e _ c o m m o n ( 1 )
lbz r _ A , 0 ( r _ a d d r )
blr
bpf_slow_path_byte_msh_neg :
lis r _ s c r a t c h1 ,- 3 2 / * S K F _ L L _ O F F * /
cmpd r _ a d d r , r _ s c r a t c h1 / * a d d r < S K F _ * * /
blt b p f _ e r r o r / * c r0 = L T * /
.globl sk_load_byte_msh_negative_offset
sk_load_byte_msh_negative_offset :
sk_ n e g a t i v e _ c o m m o n ( 1 )
lbz r _ X , 0 ( r _ a d d r )
rlwinm r _ X , r _ X , 2 , 3 2 - 4 - 2 , 3 1 - 2
blr
bpf_error_slow :
/* fabricate a cr0 = lt */
li r _ s c r a t c h1 , - 1
cmpdi r _ s c r a t c h1 , 0
bpf_error :
/* Entered with cr0 = lt */
li r3 , 0
/* Generated code will 'blt epilogue', returning 0. */
blr
